Introduction

In this chapter, we dive into artificial neural networks, one of the main drivers of artificial intelligence.

Neural networks are around since many decades. (Maybe) the first such model was built by Marvin Minsky in 1951. He called his algorithm SNARC ("stochastic neural-analog reinforcement calculator"). Since then, neural networks have gone through several stages of development. One of the milestones was the idea of Paul J. Werbos in 1974 [1] to efficiently calculate gradients in the optimization algorithm by an approach called "backpropagation". Another milestone was the use of GPUs (graphics processing units) to greatly reduce calculation time.

Artificial neural nets are extremely versatile and powerful. They can be used to

  1. fit simple models like GLMs,
  2. learn interactions and non-linear effects in an automatic way (like tree-based methods),
  3. optimize general loss functions,
  4. fit data much larger than RAM (e.g. images),
  5. learn "online" (update the model with additional data),
  6. fit multiple response variables at the same time,
  7. model input of dimension higher than two (e.g. images, videos),
  8. model input of different input dimensions (e.g. text and images),
  9. fit data with sequential structure in both in- and output (e.g. a text translator),
  10. model data with spatial structure (images),
  11. fit models with many millions of parameters,
  12. do non-linear dimension reduction.

In this chapter, we will mainly deal with the first three aspects. Since a lot of new terms are being used, a small glossary can be found in Section "Neural Network Slang".

Understanding Neural Nets

To learn how and why neural networks work, we will go through three steps - each illustrated on the diamonds data:

After this, we will be ready to build more complex models.

Step 1: Linear regression as neural net

Let us revisit the simple linear regression $$ E(\text{price}) = \alpha + \beta \cdot \text{carat} $$ calculated on the full diamonds data. In Chapter 1 we have found the solution $\hat\alpha = -2256.36$ and $\hat \beta = 7756.43$ by ordinary least-squares.

Above situation can be viewed as a neural network with

Visualized as a graph, the situation looks as follows.

Part of the figures were done with this cool webtool.

To gain confidence in neural nets, we first show that parameters estimated by a neural network are quite similar to the ones learned by linear least-squares. To do so, we will use Google's TensorFlow with its convenient (functional) Keras interface.

Example: simple linear regression

Comment: The solution of the simple neural network is indeed quite similar to the OLS solution.

The optimization algorithm

Neural nets are typically fitted by mini-batch gradient descent, using backpropagation to efficiently calculate gradients. It works as follows:

  1. Initiate the parameters with random values.
  2. Forward step: Use the parameters to predict all observations of a batch. A batch is a randomly selected subset of the full data set.
  3. Backpropagation step: Change the parameters in the right direction, making the loss of the current batch smaller. This involves calculating derivatives ("gradients") of the loss function (e.g. MSE) with respect to all parameters. Backpropagation does so in a layer-per-layer fashion, making heavy use of the chain rule.
  4. Repeat Steps 2-3 until each observation appeared in a batch. This is called an epoch.
  5. Repeat Step 4 for multiple epochs until the parameter estimates stabilize or validation performance stops improving.

Gradient descent on batches of size 1 is called "stochastic gradient descent" (SGD).

Step 2: Hidden layers

Our first neural network above consisted of only an input layer and an output layer. By adding one or more hidden layers between in- and output, the network gains additional parameters, and thus more flexibility. The nodes of a hidden layer can be viewed as latent variables, representing the original covariates. The nodes of a hidden layer are sometimes called encoding. The closer a layer is to the output, the better its nodes are suitable to predict the response variable. In this way, a neural network finds the right transformations and interactions of its covariates in an automatic way. The only ingredients are a large data set and a flexible enough network "architecture" (number of layers, nodes per layer).

Neural nets with more than one hidden layer are called "deep neural nets".

We will now add a hidden layer with five nodes $v_1, \dots, v_5$ to our simple linear regression network. The architecture looks as follows:

This network has 16 parameters. How much better than our simple network with just two parameters will it be?

Example: hidden layer

The following code is almost identical to the last one up, except that there is a hidden layer between input and output layer.

Comment: Oops, it seems as if the extra hidden layer had no effect. The reason is that a linear function of a linear function is still a linear function. Adding the hidden layer did not really change the capabilities of the model. It just added a lot of unnecessary parameters.

Step 3: Activation functions

The missing magic component is the so called activation function $\sigma$ after each layer, which transforms the values of the nodes. So far, we have implicitly used "linear activations", which - in neural network slang - is just the identity function.

Applying non-linear activation functions after hidden layers have the purpose to introduce non-linear and interaction effects. Typical such functions are

Activation functions applied to the output layer have a different purpose, namely the same as the inverse of the link function of a corresponding GLM. It maps predictions to the scale of the response:

Let us add a hyperbolic tangent activation function ($\sigma$) after the hidden layer of our simple example.

Example: activation functions

Again, the code is very similar to the last one, with the exception of using a hyperbolic tangent activation after the hidden layer (and different learning rate and number of epochs).

Comment: Adding the non-linear activation after the hidden layer has changed the model. The effect of carat is now representing the association between carat and price by a non-linear function.

Practical Considerations

Validation and tuning of main parameters

So far, we have naively fitted the neural networks without splitting the data for test and validation. Don't do this! Usually, one sets a small test dataset (e.g. 10% of rows) aside to assess the final model performance and use simple (or cross-)validation for model tuning.

In order to choose the main tuning parameters, namely

one often uses simple validation because cross-validation takes too much time.

Missing values

A neural net does not accept missing values in the input. They need to be filled, e.g., by a typcial value or a value below the minimum.

Input standardization

Gradient descent starts by random initialization of parameters. This step is optimized for standardized input. Standardization has to be done manually by either

Note that the scaling transformation is calculated on the training data and then applied to the validation and test data. This usually requires a couple of lines of code.

Categorical input

There are three ways to represent categorical input variables in a neural network.

  1. Binary and ordinal categoricals are best represented by integers and then treated as numeric.
  2. Unordered categoricals are either one-hot-encoded (i.e., each category is represented by a binary variable) or
  3. they are represented by a (categorical) embedding. To do so, the categories are integer encoded and then condensed by a special embedding layer to a few (usually 1 or 2) dense features. This requires a more complex network architecture but saves memory and preprocessing. This approach is heavily used when the input consists of words (which is a categorical variable with thousands of levels - one level per word).

For Option 2, input standardization is not required, for Option 3 it must not be applied as the embedding layer expects integers.

Callbacks

Sometimes, we want to take actions during training, such as

Such monitoring tasks are called callbacks. We will see them in the example below.

Types of layers

So far, we have encountered only dense (= fully connected) layers and activation layers. Here some further types:

Optimizer

Pure gradient descent is rarely applied without tweaks because it tends to be stuck in local minima, especially for complex networks with non-convex objective surfaces. Modern variants are "adam", "nadam" and "RMSProp". These optimizers work usually out-of-the-box, except for the learning rate, which has to be manually chosen.

Custom losses and evaluation metrics

Frameworks like Keras/TensorFlow offer many predefined loss functions and evaluation metrics. Choosing them is a crucial step, just as with tree boosting. Using TensorFlow's backend functions, one can define own metrics and loss functions (see exercises).

Overfitting and regularization

As with linear models, a model with too many parameters will overfit in an undesired way. With about 50 to 100 observations per parameter, overfitting is usually unproblematic. (For image and text data, different rules apply). Besides using less parameters, the main options to reduce overfitting are the following:

Choosing the architecture

How many layers and number of nodes per layer to select? For tabular data, using 1-3 hidden layers is usually enough. If we start with $m$ input variables, the number of nodes in the first hidden layer is usually higher than $m$ and reduces for later layers. There should not be a "representational bottleneck", i.e., an early hidden layer with too few parameters.

The number of parameters should not be too high compared to the number of rows, see "Overfitting and regularization" above.

Interpretation

Variable importance of covariates in neural networks can be assessed by permutation importance (how much performance is lost when shuffling column X?) or SHAP importance. Covariate effects can be investigated, e.g., by partial dependence plots or SHAP dependence plots.

Example: diamonds

We will now fit a neural net with two hidden layers (30 and 15 nodes) and a total of 631 parameters to model diamond prices. Learning rate and batch size were chosen by simple validation. The number of epochs is being chosen by an early stopping callback.

Comments

Example: Embeddings (optional)

Representing categorical input variables through embedding layers is extremely useful in practice. We will end this chapter with an example on how to do it with the claims data. This example also shows how flexible neural network structures are.

Exercises

  1. Fit diamond prices by gamma deviance loss with log-link (-> exponential output activation), using the custom loss function defined below. Tune the model by simple validation and evaluate it (for simplicity) on the validation dataset. Interpret the final model. Hints: I used a smaller learning rate and had to replace the "relu" activations by "tanh". Furthermore, the response needed to be transformed from int to float.
from tensorflow.keras import backend as K

def loss_gamma(y_true, y_pred):
  return -K.log(y_true / y_pred) + y_true / y_pred
  1. Study either the optional claims data example or build your own neural net, predicting claim yes/no. For simplicity, you can represent the categorical feature veh_body by integers.

Neural Network Slang

Here, we summarize some of the neural network slang.

Chapter Summary

In this chapter, we have glimpsed into the world of neural networks. Step by step we have learned how a neural network works. We have used Keras and TensorFlow to build models brick by brick.

Closing Remarks

During this lecture, we have met many ML algorithms and principles. To get used to them, the best approach is practicing. Kaggle is a great place to do so and learn from the best.

A summary and comparison of the algorithms can be found on github. Here a screenshot as per Sept. 7, 2020: .

Chapter References

[1] P.J. Werbos, "Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences", Dissertation, 1974.